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Abstract 

Taking into account the characteristics of college experimental teaching, through investigation and analysis, 
evaluation indices and an Analytical Hierarchy Process (AHP) model of experimental teaching quality have been 
established following the analytical hierarchy process method, and the evaluation indices have been given 
reasonable weights. An example is given, and the evaluation results show that the evaluation indices proposed 
in this paper are capable of reflecting objectively, exactly and reasonably experimental teaching quality, and of 
effectively promoting the quality of experimental teaching. 
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1 INTRODUCTION 

College experimental teaching plays an irreplaceable role in the cultivation of innovative talents, and 
experimental teaching quality has a direct impact on teaching quality as a whole. It is therefore necessary to 
introduce evaluations of experimental teaching quality. The key point of evaluating experimental teaching 
quality concerns how to improve it. According to the characteristics of experimental teaching, we have 
identified the core elements of experimental teaching in this paper, and provide a reasonable evaluation index 
system based on Analytical Hierarchy Process (AHP). Indices at all levels have been given reasonable weights 
based on a mathematical model, and the degree of influence each evaluation index has on experimental 
teaching quality has also been determined. Finally, the evaluation results of selected teachers are examined, 
based on a sample of evaluation data obtained using a mathematical model, which demonstrates the 
rationality and credibility of the evaluation results. 

2 AN AHP MODEL FOR EVALUATION OF EXPERIMENTAL TEACHING QUALITY 

Although the evaluation of college experimental teaching quality has been previously discussed (Chen, 2009; 
Feng, Shi & Du, 2010; Ma, Liu & Lv, 2009; Qin & Shi, 2010; Zhang, Zhou, Han & Huang, 2011), every university 
has its own features, and therefore, the evaluation index system or certain weights may not be same. This 
highlights the importance of creating an evaluation system that is suitable for each university. This paper takes 
into account the students' points of view to establish an AHP model of evaluating experimental teaching 
quality, according to the actual situation at our university, focusing on the position of the students in the 
evaluations. 
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2.1 Construction Of Evaluation System Assessing Experimental Teaching Quality 

Based on the characteristics of experimental teaching, taking into account the recommendations of experts, 
instructional supervisors, teachers and students, this evaluation system can be summarized as four aspects 
consisting of 13 factors, the hierarchical structure of which is shown in Table 1. 


Goal A 

Criterion B 

Alternative C 

Evaluation of 
experimental 
teaching quality 

Teaching attitude: B 4 

C n : Teaches and cultivates people , sets strict demands 
and is worthy of being called a teacher. 

C 12 : Engages in experimental teaching, is well-prepared 
and lectures seriously and fluently. 

C 13 : Corrects lab reports in timely, serious manner and 
patiently gives guidance. 

Teaching contents: B 2 

C 21 : Is familiar with the experimental contents and use of 
instruments; provides guidance materials 

C 22 : Contents evidence a reasonable design and are 
explained clearly and accurately. The theoretical course 
and the experimental course can be organically linked. 
The most important topics related to the subject are given 
an appropriate description. 

C 23 : The emphasis on and difficulty of the experimental 
teaching process is prominent, the amount of contents is 
suitable for students to master and the level of difficulty is 
appropriate for students to understand. 

C 24 : Comprehensively designed experiment contents are 
incorporated in the course and scientific research is 
introduced through experimental teaching. 

Teaching methods: B 3 

C 3i : Is good at inspiring students to think, stimulates the 
students' intellectual curiosity through timely guidance 
during experiments, encourages them to participate in the 
discussion of experiments and express different views. 

C 32 : Is good at guiding students as they analyze 
experimental phenomena and results, incorporating 
learned knowledge. 

C 33 : Teaching is organized in a flexible and effective 
manner, students are taught according to their aptitudes 
and instruction follows a logical order. 

Teaching results: B 4 

C 41 : Contribute to the consolidation of related theoretical 
knowledge by the students. 

C 42 : Increased knowledge, developed thinking and 
improvement in the students' practical skills through 
experimental activities. 

C 43 : Promote innovation by students and their ability to 
develop and design comprehensive experiments. 


Table 1. Evaluation system assessing experimental teaching quality 


2.2 Design Of Comparison Matrices 

Comparison matrices are the basis for weight sorting, and they have a decisive influence on the final overall 
sort. Therefore, the design of comparison matrices is a very important aspect of AHP. To accurately design 
comparison matrices at all levels, they must be carefully and objectively analyzed, researched and corrected 
until consistency verifications produce satisfactory results. To make comparisons, we need a scale of numbers 
that indicates how many times more important or dominant one element is over another with respect to the 
criterion. One common scale (Saaty, 2008) is shown in Table 2. 
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Intensity of 
importance 

Definition 

Explanation 

1 

Equal importance 

Two factors contribute equally to the 
objective. 

3 

Somewhat more important 

Experience and judgment slightly favor 
one over another. 

5 

Much more important 

Experience and judgment strongly favor 
one over another. 

7 

Very much more important 

Experience and judgment very strongly 
favor one over another. Its importance is 
demonstrated in practice. 

9 

Absolutely more important 

The evidence favoring one over the other 
is of the highest possible validity. 

2, 4, 6, 8 

Intermediate values 

Compromise is needed 

Reciprocals 
of the above 

If activity i has one of the above non-zero numbers 
assigned to it when compared to activity j, then j 
has the reciprocal value when compared to i. 



Table 2. The fundamental scale of absolute numbers 


The matrix of pairwise comparisons A = (ay) represents the intensities of the expert's preference between 
individual pairs of criteria (alternatives) (8, versus Bj, for all i, j = 1, 2,..., n). They are usually chosen according to 
a given scale (1/9, 1/8, ..., 8, 9). Given n criteria (alternatives) {B^B^..., 8„), a decision-maker compares pairs of 
criteria (alternatives) for all the possible pairs, and a comparison matrix A is thus obtained, where the element 
a-,, shows the preference weight of B, obtained by comparison with B t . 

Using the scale of relative importance shown in Table 2, a set of pairwise comparison matrices is created by 
synthesizing the recommendations of experts, instructional supervisors, teachers and students, as shown in 
Tables 3-7, where the comparison matrix A is the criteria matrix for the criteria with respect to the goal, the 
comparison matrices B lt B 2 , 8 3 , 8 4 are alternative matrices for alternatives with respect to each criterion. For 
example, we see criteria matrix A, in which the principal diagonal contains entries of 1, as each factor is of 
equal importance. The experts decide that 8 4 , teaching results, is somewhat more important than teaching 
attitude, which is rated as 3 in the cell S 4 , 8i and 1/3 in Si, S 4 . They also decide that teaching contents is slight 
more important than teaching methods, assigning a score of 2 in the cell S 2 , S 3 and 1/2 in B 3 , B 2 . The other 
elements of criteria matrix A are obtained in a similar manner, as shown below. 


A 

Bi 

b 2 

b 3 

b 4 

Priority vector W A 

Consistency check indicators 

Bi 

1 

1/4 

1/2 

1/3 

b 3 = 0.0994 

Kna, = 4.02062 

b 2 

4 

1 

2 

2 

b 2 = 0.4379 

Cl = 0.00687 

b 3 

2 

1/2 

1 

1 

b 3 = 0.2190 

CRA = 0.00764 <0.1 

b 4 

3 

1/2 

1 

1 

b 4 = 0.2437 



Table 3. Criteria matrix A and its consistency check 


Bx 

Cu 

Cl2 

Cl3 

Priority vector W Bj 

Consistency check indicators 

Cn 

1 

1/2 

1/4 

0.1429 

^max — ^ 

Cl2 

2 

1 

1/2 

0.2857 

Ch=0 

Cl3 

4 

2 

1 

0.5714 

CR 1 = 0 


Table 4. Alternative matrix B 4 and its consistency check 


b 2 

C 21 

C 2 2 

^23 

C 2 4 

Priority vector W Bj 

Consistency check indicators 

C 21 

1 

1/2 

1/4 

1/2 

0.1111 

Amax = 4 

C 22 

2 

1 

1/2 

1 

0.2222 

Cl 2 = 0 

C 23 

4 

2 

1 

2 

0.4445 

cr 2 = o 

c 24 

2 

1 

1/2 

1 

0.2222 



Table 5. Alternative matrix B 2 and its consistency check 
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b 3 

C 31 

^32 

^33 

Priority vector \N Bj 

Consistency check indicators 

C 31 

1 

4 

3 

0.6337 

A™* = 3. 0092 

C 32 

1/4 

1 

1 

0.1744 

Cl 3 = 0.0046 

C 33 

1/3 

1 

1 

0.1919 

CR 3 = 0.0079 


Table 6. Alternative matrix B 3 and its consistency check 


b 4 

C 41 

C 42 

C 43 

Priority vector Wb 4 

Consistency check indicators 

C 41 

1 

1/3 

1/5 

0.10945 

A max = 3.0037 

C 42 

3 

1 

1/2 

0.3090 

Cl 4 = 0.0018 

C 43 

5 

2 

1 

0.58155 

CR 4 = 0.0032 


Table 7. Alternative matrix B 4 and its consistency check 


2.3 Relative Weights And Consistency Check 

It is important to note that AHP does not demand perfect consistency. Some inconsistency is allowed in 
random judgments. An inconsistency ratio of about 10 percent or less is usually considered "acceptable". The 
consistency index (Cl) is calculated according to the following equation Cl = (A max -n)/(n-l), where A™* is the 
largest eigenvalue of the comparison matrix, and n is the number of criteria. The Consistency Ratio (CR) is 
calculated using the equation CR = CI/RI. The Rl is the random index representing the consistency of a randomly 
generated pairwise comparison matrix. It was derived by Saaty (1980) as average random consistency index 
(Table 8) calculated from a sample of 500 randomly generated matrices based on the AHP scale (Table 2). 


n 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Rl 

0 

0 

0.58 

0.90 

1.12 

1.24 

1.32 

1.41 

1.45 


Table 8. Random index (Rl) 


The CR tells the decision-maker how consistent he has been when making the pair-wise comparisons. If 
CR<0.10, the decision-maker's pair-wise comparisons are relatively consistent and the criterion is considered to 
have acceptable consistency. If CR > 0.10, the decision-maker should seriously consider re-evaluating his pair¬ 
wise comparisons - the sources of inconsistency must be identified and resolved and the matrix reanalyzed. 
The priority vector WA (weights of the criteria corresponding to the goal) is the normalizing eigenvector 
corresponding to A max of matrix A, and the consistency ratio of matrix A is CR A = 0.00764, as shown in last two 
columns of Table 3, respectively. Similarly, the priority vectors W B , Wb 2 , Wb 3 , Wb 4 (weights of alternatives for 
each criterion) and their consistency check indicators are listed in last two columns of Tables 4-7, respectively. 

2.4 Combinatorial Weights And Combinatorial Consistency Check 

The priorities of each alternative for the goal are referred to as combinatorial weights, and the components of 
the combinatorial weight vector are calculated as described by Han (2005): 


4 

1/1/, = ^ bjCij (/ =1,... ,13) , where each value of b, and c, y is listed in Table 9. 

i= i 


The combinatorial consistency ratio is calculated according to the formula: 


CR = CR A +Y t bjC\ J /t t bp\, 

F 1 / ;=1 

CR = 0.00764 + (0.0994 x 0 + 0.4379 x 0 + 0.2190 x 0.0046 + 0.2437 x 0.0018)/(0.0994 x 0.58 + 0.4379 x 0.9 + 
0.219 x 0.58 + 0.2437 x 0.58) = 0.0096 < 0.1; therefore, the combinatorial consistency is acceptable, and the 
result of the global ranking has satisfactory consistency. If CR > 0.10, the decision-maker should seriously 
reconsider the model or reconstruct the comparison matrices so that they have a higher consistency ratio. 


X 
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Alternative C 

Criterion B 

Combinatorial 
weights w, 

Bi 

b 2 

Ba 

b 4 

b, = 0.0994 

b 2 = 0.4379 

b 3 = 0.2190 

b , = 0.2437 

Cn 

Cn = 0.1429 

Cn = 0 

C13 = 0 

Ci 4 = 0 

Wi= 0.0142 

Cl 2 

0.2857 

0 

0 

0 

w 2 = 0.0284 

Cl 3 

0.5714 

0 

0 

0 

w 3 = 0.0568 

C2I 

0 

0.1111 

0 

0 

w 4 = 0.0487 

C22 

0 

0.2222 

0 

0 

w 5 = 0.0973 

C23 

0 

0.4445 

0 

0 

w 6 = 0.1946 

C24 

0 

0.2222 

0 

0 

w 7 = 0.0973 

C3I 

0 

0 

0.6337 

0 

Wg — 0.1388 

C32 

0 

0 

0.1744 

0 

w 9 = 0.0382 

C33 

0 

0 

0.1919 

0 

W10 = 0.0420 

C41 

0 

0 

0 

0.10945 

Wn = 0.0267 

C 42 

0 

0 

0 

0.3090 

w i2 = 0.0753 

C43 

0 

0 

0 

0.58155 

W13 = 0.1417 


Table 9. Combinatorial weights 


2.5 Model Evaluation 

As presented in Table 3, it was observed that for students, "teaching contents" represented the most 
important criterion, followed closely by "teaching results" and "teaching methods". It can be seen that 
students rank teaching results higher than teaching methods. Lastly, "teaching attitude" does not seem to be 
particularly important to students. This model is in agreement with the current situation at engineering 
universities, and it comprehensively reflects student evaluations of experimental teaching quality. If a teacher 
does not have a certain amount of theoretical knowledge and practical experience, he can not reconcile theory 
with practice in the process of teaching, his instructional process is more boring, and is not very effective. 

3 APPLICATION OF THE AHP MODEL 

We wrote a questionnaire that requires students to evaluate their teachers, assigning them one of five grades 
(A, B, C, D and E; equivalent to very satisfied, satisfied, generally, dissatisfied, very dissatisfied) for each of the 
13 alternatives shown in Table 1. Table 10 shows the evaluation results of 105 students for three teachers. 

First, the statistical data are quantified with the values 5, 4, 3, 2, 1, which correspond to the ranking grades A, 
B, C, D and E. Accordingly, taking into account the membership function of the Cauchy distribution: 

[olnx + ti, 3<x<5, 

Where a, (5, a, b are constants to be determined. Supposing that the membership degree is 1 when the 
evaluation is grade A, this means that/f5j = 1; the membership degree is 0.70 when the evaluation is grade C ( 
f(3) = 0.70); and the membership degree is 0.10 when the evaluation is grade E (f( 1) = 0.1). The values of a, /?, 
a, b are then determined to be a = 2.8049, /? = 0.4417, a = .5873 and b = 0.0548, respectively. The membership 
function is obtained by substituting the values of a, /?, a, b in the formula (*): 

r , , f[l + 2.8049(x- 0.4417)' 2 ]’ 1 , 1 < x < 3, 

f(x) = i 

[0.5873lnx +0.0548, 3<x<5. 

The values of the membership function at x = 2, 4 are calculated as /(2) = 0.4640, /(4) = 0.8690, thus grades A, 
B, C, D and E are quantified with values {1, 0.869, 0.70, 0.464, 0.1). The scores for each alternative for each 
teacher are calculated using the data in Table 10, listed in columns 3, 5 and 7. The total scores for each teacher 
are obtained by adding the products of each alternative score and its combinatorial weight w ,• obtained from 
Table 9, shown in the last row of Table 10 as the evaluation result tallies for the current situation. 
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Alternative 

Teacher 1 

Score 

Teacher 2 

Score 

Teacher 3 

Score 

C n 

68A 30B 6C1D 

0.9403 

58A 28B 15C 

4D 

0.9027 

53A 38B 6C 

7D1E 

0.8911 

Ci2 

67A31B2C5D 

0.9301 

30A 45B 21C 

5D4E 

0.8240 

52A 40B 8C 

5D 

0.9017 

Cl 3 

71A 17B15C2D 

0.9257 

43A 13B 25C 

16D 8E 

0.7621 

52A 32B 16C 

5D 

0.8888 

Ql 

60A30B 12C 3D 

0.9130 

38A27B17C 

14D 9E 

0.7691 

48A 43B 11C 

3D 

0.8996 

C 2 2 

65A 36B 4C 

0.9437 

41A 36B 24C 

4D 

0.8661 

40A 51B 9C 

5D 

0.8851 

C23 

69A 16B 9C 7D 

4E 

0.8843 

33A 45B 22C 

5D 

0.8555 

51A 33B 14C 

7D 

0.8831 

C24 

68A33B2C2D 

0.9429 

36A 36B 25C 

6D2E 

0.8359 

35A47B17C 

4D 2E 

0.8552 

C31 

71A 23B 5C 6D 

0.9264 

43A38B17C 

4D3E 

0.8579 

65A 23B 13C 

4D 

0.9137 

C32 

63A 27B 8C 7D 

0.9077 

31A47B21C 

6D 

0.8507 

40A 43B IOC 

9D 3E 

0.8461 

C33 

65A21B 17C2D 

0.9150 

34A 45B17C 

3D 6E 

0.8285 

45A 33B 18C 

5D4E 

0.8476 

C41 

62A 26B 13C 4D 

0.9100 

40A 37B 21C 

7D 

0.8581 

49A 28B 23C 

5D 

0.8738 

C42 

69A23B 9C4D 

0.9252 

46A 35B 18C 

5D1E 

0.8708 

43A 26B 28C 

4D 4E 

0.8329 

C43 

57A28B 13C5D 

2E 

0.8853 

31A 33B 26C 

10D 5E 

0.7906 

38A 30B 20C 

12D5E 

0.8013 

Total score 


0.9136 


0.8359 


0.8681 


Table 10. Statistical data on experimental teaching quality 


4 CONCLUSION 

The evaluation indicators and their weights of experimental teaching quality are developed according to the 
AHP method. As compared to other methods, the greatest advantage of the analytical hierarchy process (AHP) 
is that it is able to combine both qualitative and quantitative methods and to consider all influencing factors as 
fully as possible. The evaluation results of the AHP method are more objective, scientific and rational, and it has 
been proven that the AHP method is the most appropriate when surveys need to account for a high degree of 
intuition and subjectivity. The evaluation results for the 3 teachers in section 3 are only from a student 
perspective; if we administer the same questionnaire to experts, instructional supervisors and colleagues of the 
3 teachers, requesting their evaluations of the 3 teachers according to the 13 alternatives in Table 1, it would 
be possible to obtain evaluation results for the 3 teachers from the experts' points of view, using the same AHP 
model developed in this paper. This would provide more objective evaluations for the 3 teachers, synthesizing 
the students' and the experts' evaluation results. The model established in this paper can be extended to the 
comprehensive evaluation of textbooks, teaching management, quality courses, and other situations requiring 
comprehensive evaluation. The AHP method has already been used in many applications (Saaty, 2008; ISAHP, 
2009). 
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